{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Log output\n",
    "\n",
    "We use the same example as in the quickstart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "using MIPVerify\n",
    "using Gurobi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnist = MIPVerify.read_datasets(\"MNIST\")\n",
    "n1params = MIPVerify.get_example_network_params(\"MNIST.n1\")\n",
    "sample_image = MIPVerify.get_image(mnist.train.images, 1)\n",
    "target_label_index = 9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Viewing Log Output\n",
    "\n",
    "Our package logs to STDOUT at the `debug`, `info`, `notice` and `warn` levels. \n",
    "The default output level is to show logs only at the `notice` level and above, which is also the recommended level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[36m[notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit. The model built will be cached and re-used for future solves, unless you explicitly set rebuild=false.\n",
      "\u001b[39m\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9]\n",
      "\u001b[39mOptimize a model with 3385 rows, 3256 columns and 71132 nonzeros\n",
      "Variable types: 3196 continuous, 60 integer (60 binary)\n",
      "Coefficient statistics:\n",
      "  Matrix range     [2e-05, 7e+02]\n",
      "  Objective range  [1e+00, 1e+00]\n",
      "  Bounds range     [1e+00, 1e+02]\n",
      "  RHS range        [1e-02, 7e+02]\n",
      "Presolve removed 2860 rows and 2184 columns\n",
      "Presolve time: 0.11s\n",
      "Presolved: 525 rows, 1072 columns, 65472 nonzeros\n",
      "\n",
      "MIP start did not produce a new incumbent solution\n",
      "MIP start violates constraint R1024 by 1.000000000\n",
      "\n",
      "Variable types: 1012 continuous, 60 integer (60 binary)\n",
      "\n",
      "Root relaxation: objective 0.000000e+00, 242 iterations, 0.01 seconds\n",
      "\n",
      "    Nodes    |    Current Node    |     Objective Bounds      |     Work\n",
      " Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time\n",
      "\n",
      "     0     0    0.00000    0    9          -    0.00000      -     -    0s\n",
      "Another try with MIP start\n",
      "H    0     0                      55.3332559    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0   10   55.33326    0.00000   100%     -    0s\n",
      "H    0     0                      27.1696380    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0   10   27.16964    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0   11   27.16964    0.00000   100%     -    0s\n",
      "H    0     0                       9.8928387    0.00000   100%     -    0s\n",
      "H    0     0                       9.8622769    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "     0     2    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "*  354   156              25       9.7569972    0.00000   100%  68.9    1s\n",
      "  4777   468    8.90557   19   10    9.75700    4.59017  53.0%  58.4    5s\n",
      "\n",
      "Cutting planes:\n",
      "  Gomory: 3\n",
      "  Cover: 3\n",
      "  MIR: 7\n",
      "  Flow cover: 10\n",
      "\n",
      "Explored 6902 nodes (388477 simplex iterations) in 6.66 seconds\n",
      "Thread count was 8 (of 8 available processors)\n",
      "\n",
      "Solution count 5: 9.757 9.86228 9.89284 ... 55.3333\n",
      "Pool objective bound 9.757\n",
      "\n",
      "Optimal solution found (tolerance 1.00e-04)\n",
      "Best objective 9.756997241475e+00, best bound 9.756997241475e+00, gap 0.0000%\n"
     ]
    }
   ],
   "source": [
    "d = MIPVerify.find_adversarial_example(n1params, sample_image, target_label_index, GurobiSolver(), rebuild=true);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "More information is available if we set the log level to `debug`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[36m[notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit. The model built will be cached and re-used for future solves, unless you explicitly set rebuild=false.\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 0.0\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 236.19499039114618\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 329.4706219653086\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 347.3507384582369\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 490.3354319365959\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 111.66031695101941\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 230.85744098698652\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 110.09320929282381\n",
      "\u001b[39m\u001b[32m[info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.5778275495135097.\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 250.1943900717896\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 263.9686250131348\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 443.7377089444059\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 170.58623081063135\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 546.9967283877565\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 342.81111856390635\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 561.8499789605797\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 322.1609214959067\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 252.888615674684\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 190.94181084213062\n",
      "\u001b[39m\u001b[32m[info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.9188018145812635.\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 389.16818001562916\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 120.5688694681638\n",
      "\u001b[39m\u001b[32m[info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.43244015627890753.\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 314.1279773027621\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 333.3978216299831\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 395.66929790133076\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 156.25265560651098\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 263.63110342096843\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 306.08282612869147\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 259.3597215144569\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 176.38206258926618\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 279.1469821915614\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 301.9881744933099\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 363.84051783382193\n",
      "\u001b[39m\u001b[32m[info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.4156007693486492.\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 306.02212780906495\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 223.20192365427127\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 146.17861158898927\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 397.6660964044127\n",
      "\u001b[39m\u001b[32m[info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.23529601270679312.\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 278.0674748427061\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 354.40977058244283\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 184.93657306973148\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 347.3454124031803\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δu = 256.17035614083665\n",
      "\u001b[39m\u001b[34m[debug | MIPVerify]:   Δl = 540.5128944047548\n",
      "\u001b[39m\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9]\n",
      "\u001b[39mOptimize a model with 3385 rows, 3256 columns and 71132 nonzeros\n",
      "Variable types: 3196 continuous, 60 integer (60 binary)\n",
      "Coefficient statistics:\n",
      "  Matrix range     [2e-05, 7e+02]\n",
      "  Objective range  [1e+00, 1e+00]\n",
      "  Bounds range     [1e+00, 1e+02]\n",
      "  RHS range        [1e-02, 7e+02]\n",
      "Presolve removed 2860 rows and 2184 columns\n",
      "Presolve time: 0.13s\n",
      "Presolved: 525 rows, 1072 columns, 65472 nonzeros\n",
      "\n",
      "MIP start did not produce a new incumbent solution\n",
      "MIP start violates constraint R1024 by 1.000000000\n",
      "\n",
      "Variable types: 1012 continuous, 60 integer (60 binary)\n",
      "\n",
      "Root relaxation: objective 0.000000e+00, 242 iterations, 0.01 seconds\n",
      "\n",
      "    Nodes    |    Current Node    |     Objective Bounds      |     Work\n",
      " Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time\n",
      "\n",
      "     0     0    0.00000    0    9          -    0.00000      -     -    0s\n",
      "Another try with MIP start\n",
      "H    0     0                      55.3332559    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0   10   55.33326    0.00000   100%     -    0s\n",
      "H    0     0                      27.1696380    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0   10   27.16964    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0   11   27.16964    0.00000   100%     -    0s\n",
      "H    0     0                       9.8928387    0.00000   100%     -    0s\n",
      "H    0     0                       9.8622769    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "     0     2    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "*  839   309              27       9.7569972    0.00000   100%  70.8    1s\n",
      "  4504   347    5.30680   13   13    9.75700    5.30680  45.6%  62.8    5s\n",
      "\n",
      "Cutting planes:\n",
      "  Gomory: 3\n",
      "  Cover: 2\n",
      "  MIR: 7\n",
      "  Flow cover: 11\n",
      "\n",
      "Explored 6030 nodes (364445 simplex iterations) in 6.24 seconds\n",
      "Thread count was 8 (of 8 available processors)\n",
      "\n",
      "Solution count 5: 9.757 9.86228 9.89284 ... 55.3333\n",
      "Pool objective bound 9.757\n",
      "\n",
      "Optimal solution found (tolerance 1.00e-04)\n",
      "Best objective 9.756997241475e+00, best bound 9.756997241475e+00, gap 0.0000%\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dict{Symbol,Any} with 7 entries:\n",
       "  :PerturbationParameters => additive\n",
       "  :TargetIndexes          => [9]\n",
       "  :SolveStatus            => :Optimal\n",
       "  :Output                 => JuMP.GenericAffExpr{Float64,JuMP.Variable}[-0.0120…\n",
       "  :Model                  => Minimization problem with:…\n",
       "  :Perturbation           => JuMP.Variable[__anon__ __anon__ __anon__ __anon__ …\n",
       "  :PerturbedInput         => JuMP.Variable[__anon__ __anon__ __anon__ __anon__ …"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MIPVerify.setloglevel!(\"debug\")\n",
    "d = MIPVerify.find_adversarial_example(n1params, sample_image, target_label_index, GurobiSolver(), rebuild=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "When we select `rebuild=false`, we do not do any of the work in determining upper and lower bounds, and none of the `info` and `debug` logs are produced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[36m[notice | MIPVerify]: Loading model from cache.\n",
      "\u001b[39m\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9]\n",
      "\u001b[39mOptimize a model with 3385 rows, 3256 columns and 71132 nonzeros\n",
      "Variable types: 3196 continuous, 60 integer (60 binary)\n",
      "Coefficient statistics:\n",
      "  Matrix range     [2e-05, 7e+02]\n",
      "  Objective range  [1e+00, 1e+00]\n",
      "  Bounds range     [1e+00, 1e+02]\n",
      "  RHS range        [1e-02, 7e+02]\n",
      "Presolve removed 2860 rows and 2184 columns\n",
      "Presolve time: 0.13s\n",
      "Presolved: 525 rows, 1072 columns, 65472 nonzeros\n",
      "\n",
      "MIP start did not produce a new incumbent solution\n",
      "MIP start violates constraint R1024 by 1.000000000\n",
      "\n",
      "Variable types: 1012 continuous, 60 integer (60 binary)\n",
      "\n",
      "Root relaxation: objective 0.000000e+00, 242 iterations, 0.01 seconds\n",
      "\n",
      "    Nodes    |    Current Node    |     Objective Bounds      |     Work\n",
      " Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time\n",
      "\n",
      "     0     0    0.00000    0    9          -    0.00000      -     -    0s\n",
      "Another try with MIP start\n",
      "H    0     0                      55.3332559    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0   10   55.33326    0.00000   100%     -    0s\n",
      "H    0     0                      27.1696380    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0   10   27.16964    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0   11   27.16964    0.00000   100%     -    0s\n",
      "H    0     0                       9.8928387    0.00000   100%     -    0s\n",
      "H    0     0                       9.8622769    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "     0     0    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "     0     2    0.00000    0    1    9.86228    0.00000   100%     -    0s\n",
      "*  839   309              27       9.7569972    0.00000   100%  70.8    1s\n",
      "  4504   347    5.30680   13   13    9.75700    5.30680  45.6%  62.8    5s\n",
      "\n",
      "Cutting planes:\n",
      "  Gomory: 3\n",
      "  Cover: 2\n",
      "  MIR: 7\n",
      "  Flow cover: 11\n",
      "\n",
      "Explored 6030 nodes (364445 simplex iterations) in 6.24 seconds\n",
      "Thread count was 8 (of 8 available processors)\n",
      "\n",
      "Solution count 5: 9.757 9.86228 9.89284 ... 55.3333\n",
      "Pool objective bound 9.757\n",
      "\n",
      "Optimal solution found (tolerance 1.00e-04)\n",
      "Best objective 9.756997241475e+00, best bound 9.756997241475e+00, gap 0.0000%\n"
     ]
    }
   ],
   "source": [
    "d = MIPVerify.find_adversarial_example(n1params, sample_image, target_label_index, GurobiSolver(), rebuild=false);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Handling log output\n",
    "\n",
    "We use the logger provided by the [`Memento`](https://github.com/invenia/Memento.jl) package. Rather than just having logs written to STDOUT, you can choose to capture the output (for example, to write logs to file) by adding your own handlers to `MIPVerify.LOGGER`. See the Memento documentation for more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Memento.Logger"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "typeof(MIPVerify.LOGGER)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 0.6.0",
   "language": "julia",
   "name": "julia-0.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
